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Congressional  Committees 

The  Department  of  the  Army  began  a  pilot  project  in  July  1997  to  test  an 
alternative  approach  for  providing  relocation  services  for  its  military 
personnel  stationed  at  Hunter  Army  Airfield,  Savannah,  Georgia.  The  Army 
undertook  this  effort  to  address  long-standing  concerns  and  problems 
associated  with  the  current  personal  property  program.  The  Department  of 
Defense  (DOD)  has  two  other  pilot  programs  underway  to  test  different 
approaches  to  improving  its  personal  property  program  and  is  proposing  a 
fourth  pilot.  To  determine  which  pilot,  or  portions  thereof,  could  provide 
better  long-term  results,  the  Deputy  Under  Secretary  of  Defense  (Logistics) 
tasked  the  U.S.  Transportation  Command  (USTRANSCOM)  to  oversee  all 
personal  property  pilot  tests  and  recommend  the  follow-on  course  of 
action. 

The  statement  of  managers  in  the  conference  report  on  the  1997  DOD 
Appropriations  Act  directed  us  to  validate  the  results  and  savings  achieved 
before  DOD  expands  any  of  its  personal  property  pilot  projects.1  In  this 
regard,  we  testified  in  March  1999  on  the  status  of  the  Army’s  pilot  and 
DOD’s  efforts  to  improve  its  personal  property  program.2  This  report 
supplements  our  testimony  and  principally  addresses  the  results  of  our 
review  of  the  12-month  Hunter  pilot  test  and  lessons  for  evaluating  other 
pilots.  Our  objectives  were  to  (1)  assess  the  Army’s  evaluation 
methodology  of  the  Hunter  pilot,  including  the  validity  of  data  and  reported 
results  and  (2)  determine  the  status  of  all  ongoing  and  planned  pilot 
projects  and  the  adequacy  of  DOD’s  plans  to  evaluate  the  pilot  projects. 


Results  in  Brief  Although  the  Army  reported  that  the  Hunter  pilot  was  a  success,  we  found 

'  that  most  of  the  results  of  its  evaluation  were  inconclusive.  While  the 

Hunter  pilot  provides  services  and  benefits  that  were  not  previously 
available  during  the  moving  process,  we  were  unable  to  validate  all 


‘House  Report  104-863  (Sept.  28,  1996)  p.  865. 

2 Defense  Transportation:  Efforts  to  Improve  DOD’s  Personal  Property  Program  (GAO/T-NSIAD-99-106, 
Mar.  18, 1999). 
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reported  results  of  the  Army’s  evaluation  of  this  pilot  because  of 
weaknesses  in  the  evaluation  methodology  and  data.  Specifically,  because 
of  the  methods  used  in  conducting  the  customer  surveys,  we  could  not 
confirm  that  customer  satisfaction  improved.  Also,  due  to  weaknesses  in 
the  Army’s  methodology  and  data  reliability,  we  could  not  validate  the 
extent  to  which  pilot  costs  exceeded  baseline  costs.  However,  we  were 
able  to  confirm  that  33  percent  of  the  pilot  shipments  were  awarded  to 
small  business  carriers  and  agents.  Further,  lessons  learned  by  the  Army  in 
developing  an  evaluation  plan,  conducting  the  pilot  test,  and  evaluating 
results  can  provide  useful  information  to  DOD  as  it  conducts  and  assesses 
other  pilot  efforts. 

The  Navy  and  the  Military  Traffic  Management  Command  (MTMC)  each 
have  a  personal  property  pilot  project  underway,  and  DOD  is  proposing  a 
fourth  pilot  to  test  different  approaches  to  improve  its  personal  property 
program.  As  a  result,  DOD  will  be  running  multiple  pilots  concurrently, 
with  different  goals,  objectives,  and  expected  outcomes.  USTRANSCOM3  is 
tasked  with  evaluating  the  results  of  the  pilots  and  using  that  information 
to  recommend  a  redesigned  Department-wide  relocation  program. 
However,  DOD  has  not  yet  determined  how  many  approaches  will 
ultimately  be  tested  and  the  milestones  for  completing  the  pilots’ 
evaluation  and  implementing  an  improved  process,  nor  has  it  assured  itself 
that  a  methodologically  sound  evaluation  process  is  in  place  to  execute  this 
process. 

Improving  DOD’s  personal  property  program  has  been  a  slow,  complex 
process.  DOD  and  the  services  have  spent  a  large  amount  of  time  and  effort 
to  dramatically  change  the  quality  of  services  military  personnel  receive. 
We  support  pilots  as  a  tool  to  test  different  approaches.  However,  before 
the  Department  can  make  any  credible  decisions  on  changing  the  current 
program,  we  recommend  that  the  Secretary  of  Defense  direct 
USTRANSCOM  to  develop  a  comprehensive  strategy  for  evaluating  each  of 
the  pilots’  attributes  in  a  comparable  manner.  Further,  on  the  basis  of  the 
Army’s  lessons  learned  in  conducting  the  Hunter  pilot  and  of  our  evaluation 
of  the  pilot,  we  recommend  that  the  Department  seek  expert 
methodological  advice  to  enhance  the  quality  of  its  assessment. 


3The  mission  of  USTRANSCOM,  which  is  DOD's  single  manager  of  all  Defense  transportation  services, 
is  to  provide  global  air,  land,  and  sea  transportation  to  meet  national  security  needs.  USTRANSCOM 
executes  its  mission  through  three  component  commands:  MTMC  for  land  transportation  and  port 
operations,  the  Military  Sealift  Command  for  sea  transportation,  and  the  Air  Mobility  Command  for  air 
transportation. 
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Background 


DOD  has  long  been  concerned  about  the  quality  of  service  it  provides 
military  personnel  and  their  families  when  they  relocate.  DOD  spends 
approximately  $3  billion  annually  to  transport,  store,  and  manage  the 
household  goods  and  unaccompanied  baggage  of  its  servicemembers  and 
families.  Past  problems  included  poor  service  from  movers,  excessive 
incidence  of  loss  or  damage  to  servicemembers’  property,  and  high  claims 
costs  to  the  government.  All  of  these  problems  contributed  to  poor  quality 
of  service  for  people  using  the  system. 


Most  Results 
Army  Hunter 
Inconclusive 


of  the 
Pilot  Are 


Because  of  weaknesses  in  the  Army’s  evaluation  methodology  and  data,  we 
were  unable  to  validate  the  reported  results  of  the  quality  of  life  and  cost 
factors  of  the  Hunter  pilot  program.  Moreover,  the  Army’s  evaluation  plan 
methodology  was  not  an  effective  tool  for  collecting  and  analyzing  the  pilot 
results.  However,  lessons  learned  by  the  Army  in  conducting  the  12-month 
pilot  test  do  provide  useful  information  to  DOD  as  it  conducts  and  assesses 
its  pilot  efforts.  Also,  the  Hunter  pilot  is  providing  services  and  benefits 
that  were  not  previously  available,  including  point-to-point  move 
management,  personal  move  counseling  and  coordination,  direct  claims 
settlement,  assistance  in  buying/selling  a  residence,  and  visibility  of  the 
shipment  throughout  the  move. 


The  Army’s  Assessment  and  The  Army  reported  that  through  its  first  12  months  of  operations,  the 
Evaluation  Annroach  Hunter  pilot  successfully  demonstrated  that  commercial  practices  could  be 

*  ^  applied  to  the  military  relocation  process.  Specifically,  the  Army  was  able 

to  contract  with  Cendant  Mobility,  a  move  management  company,  to 
provide  services  similar  to  those  available  in  the  private  sector.  In  January 
1997,  the  Army  began  developing  a  test  evaluation  plan  that  defined  roles 
and  responsibilities,  test  factors,  and  processes  for  capturing  data  related 
to  each  test  factor  and  stated  how  test  data  would  be  compared  to  that 
from  the  existing  program  (i.e.,  the  baseline) .  The  plan  stated  that  the  Army 
Audit  Agency  (AAA)  would  be  responsible  for  validating  the  baseline  and 
test  data  and  for  providing  the  Army  a  report  on  the  test  results.  We 
reported  in  June  1998  that  the  Army  needed  to  further  define  the  various 
factors  and  measurements  to  be  included  in  its  evaluation  plan.4  The  Army 
clarified  how  it  would  measure  pilot  success  and  revised  its  methodology 


4 Defense  Transportation:  The  Army’s  Hunter  Pilot  Project  to  Outsource  Relocation  Services 
(GAO/NSIAD-98-149,  June  10, 1998). 
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in  the  evaluation  plan,  dated  August  12, 1998.  This  represented  the  third 
such  revision,  which  was  not  completed  until  after  the  12-month  test 
period. 

The  modified  plan  indicated  that  the  pilot  results  would  be  reviewed  on  the 
basis  of  three  factors — quality  of  life,  cost,  and  impact  on  small  business — 
and  would  use  a  5-point  scoring  system.  The  Army  assigned  the  greatest 
weight  to  the  quality  of  life  factor,  which  consisted  of  one  point  each  for 
three  sub-factors— customer  satisfaction,  average  claims  settlement  time, 
and  percentage  of  direct  deliveries.  Along  with  the  3  points  for  quality  of 
life,  the  Army  assigned  1  point  each  for  two  other  factors,  total  cost  and 
impact  on  small  business,  for  a  total  of  5  points.  Appendix  I  provides 
information  on  the  scale  and  scores  used  by  the  Army  to  evaluate  the 
results  of  each  factor  and  sub-factor. 

In  its  October  1998  evaluation  report  and  February  and  March  1999 
supplements,  the  Army  reported  to  us  that  Cendant  Mobility  managed 
1,349  shipments  involving  household  goods,  unaccompanied  baggage, 
do-it-yourself  (DITY)  moves,  and  mobile  homes  during  the  12-month  test. 
(See  appendix  II  for  the  number  of  shipments  in  each  category,  including 
domestic  and  international  volumes.)  The  Army  also  reported  that  the  pilot 
project  earned  a  passing  score  on  each  of  the  test  factors  that  the  Army 
measured  and  deemed  the  Hunter  pilot  a  success  because  the  factor  points 
exceeded  the  minimum  threshold  of  3  based  on  a  5-point  scale.  According 
to  the  Army’s  analysis,  the  Hunter  pilot  test  earned  3.75  points  as  follows: 

•  customer  satisfaction  improved  by  over  1 1  percent  (0.25  point) ; 

•  the  initial  offer  to  settle  claims  averaged  9  days  (1  point) ; 

•  100  percent  of  eligible  shipments  were  delivered  directly  to  the 
servicemember  without  storage  in  transit  (1  point); 

•  pilot  project  costs  exceeded  baseline  costs  by  18.6  percent  (0.50  point); 
and 

•  33  percent  of  the  shipments  were  awarded  to  small  businesses  (1  point) . 

The  Army  stated  that  it  also  considered  other  factors  generally  related  to 
transportation  process  improvements  and  did  not  rely  on  test  scores  alone 
to  determine  the  pilot  a  success.  These  factors  included  simplifying  the 
price  structure  and  claims  process,  providing  pre-audit  services,  and 
minimizing  government-unique  requirements.  Each  servicemember  worked 
with  one  personal  move  coordinator  who  integrated  the  member’s  move, 
providing  point-to-point  move  management,  coordination,  and  prompt 
resolution  of  all  problems.  This  was  in  contrast  to  the  current  process,  in 
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which  a  servicemember  may  need  to  contact  four  or  five  different  offices 
over  the  course  of  a  single  move.  Neither  AAA  nor  we  reviewed  these  cited 
benefits  because  they  were  not  included  as  measurable  factors  in  the 
evaluation  plan. 


Evaluation  Weaknesses  Led  We  identified  a  number  of  shortfalls  in  the  Army’s  evaluation  methodology 
to  Inconclusive  Results  and  data.  Specifically,  the  methodology  for  surveying  customer  satisfaction 

and  calculating  claims  settlement  time,  percentage  of  direct  deliveries 
made,  and  overhead  costs  did  not  allow  the  Army  to  obtain  and  analyze  the 
data  needed  to  accurately  assess  pilot  results.  Furthermore,  in  several 
instances  the  data  collected  to  support  the  measurements  did  not 
demonstrate  achievement  of  the  Army’s  reported  results.  Most  importantly, 
we  could  not  confirm  that  customer  satisfaction  improved  by  11.5  percent. 
Further,  we  could  not  confirm  that  all  eligible  shipments  were  delivered 
directly  without  incurring  in-transit  storage  costs,  offer  of  claims 
settlement  time  averaged  9  days,  and  the  pilot  program  cost 
18.6  percent  more  than  the  baseline  cost.  On  the  other  hand,  we  were  able 
to  confirm  that  33  percent  of  the  shipments  were  awarded  by  Cendant 
Mobility  to  small  business  carriers  and  agents — 10  percent  over  the  Army’s 
minimum  threshold. 


Inconclusive  Results  for  The  Army’s  methodology  for  evaluating  customer  satisfaction  results  was 

Customer  Satisfaction  flawed  as  it  allowed  servicemembers  to  be  surveyed  multiple  times — by 

Cendant  Mobility,  two  survey  contractors,  and  carriers.  In  addition  to  the 
use  of  different  populations  and  respondent  rates,  the  various  survey 
instruments  also  contained  different  questions  used  to  measure  the  level  of 
customer  satisfaction.  Therefore,  we  could  not  validate  that  customer 
satisfaction  improved  by  1 1.5  percent,  as  reported  by  the  Army,  on  the  basis 
of  the  conclusions  reached  by  one  of  the  four  survey  instruments. 
Notwithstanding  these  concerns,  other  data  gathered  in  these  surveys  but 
not  scored  by  the  Army  in  accordance  with  its  evaluation  plan,  provided 
indicators  of  customer  satisfaction  as  well  as  lessons  learned  that  may  be 
useful  to  DOD  for  conducting  other  pilots. 

To  measure  customer  satisfaction,  the  Army  directed  its  contractor 
(Battelle’s  Pacific  Northwest  National  Laboratory)  to  survey  members  at 
Hunter  and  those  making  similar  moves  at  Fort  Stewart,  Georgia  (the 
baseline).  Responses  from  servicemembers  at  both  locations  would  be 
compared.  Subsequently,  the  Army  decided  to  use  USTRANSCOM’s  survey, 
conducted  by  Electronic  Data  Systems  (EDS),  to  evaluate  and  score  the 
pilot  project. 
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Nonetheless,  servicemembers  were  surveyed  multiple  times,  possibly  by  as 
many  as  four  different  entities.  EDS  and  Battelle  (the  survey  contractors), 
Cendant  Mobility  (which  was  contractually  required  to  survey  a  sample  of 
customers),  and  the  carrier  (for  its  own  quality  control  purposes)  each  may 
have  contacted  the  same  servicemembers  to  query  them  on  their  move 
experiences.  As  a  result,  the  customer  satisfaction  results  were 
compromised  and  therefore  inconclusive.  EDS  reported  that  some 
servicemembers  refused  to  respond  to  the  survey  and  that  an 
undetermined  number  who  did  respond  gave  perfunctory  answers.  In  a 
caveat  in  its  report,  Battelle  stated  that  it  was  aware  that  some  individuals 
were  interviewed  multiple  times. 

The  Army  informed  us  that  it  did  not  use  Battelle ’s  results  because  the 
Army  found  the  draft  report  difficult  to  interpret  and  inconclusive.  The 
Army  also  had  concerns  about  a  disclaimer  on  the  survey  results.5 
Moreover,  the  contractors’  survey  results  varied  significantly.  While  EDS 
reported  that  11.5  percent  more  of  the  Hunter  respondents  were  satisfied 
with  their  current  relocation  experience  than  the  Fort  Stewart 
respondents,  Battelle  reported  only  a  3-percent  difference  between  the  two 
groups  of  respondents.6  We  cannot  explain  all  the  reasons  for  these 
differences.  We  do  know  that  the  questions  asked  by  each  contractor  to 
determine  customer  satisfaction  were  not  identical  and  that  the  population 
and  number  of  respondents  in  the  two  surveys  varied  significantly.  EDS 
also  had  a  much  lower  response  rate  (53  percent)  than  Battelle 
(89  percent).  Further,  EDS  used  a  75-percent  significance  level,  while 
Battelle  based  its  results  using  a  95-percent  significance  level.  The  latter  is 
the  standard  used  in  social  science  research.7 

Other  data  obtained  by  the  surveys  but  not  scored  by  the  Army,  in 
accordance  with  its  evaluation  plan,  provides  indicators  of  customer 


5The  Army  later  learned  that  the  Department  of  Energy  requires  such  a  disclaimer  in  all  studies 
performed  for  government  entities  and  that  the  disclaimer  had  no  relevance  to  the  results  reported  by 
Battelle. 

6Battelle  also  reported  that  the  3-percent  figure  was  "statistically  insignificant,”  meaning  that  the 
variance  in  responses  from  the  two  groups  (personnel  at  Hunter  and  at  Fort  Stewart)  was  not  great 
enough  to  conclude  that  the  move  experiences  differed  significantly. 

Significance  level  is  the  likelihood  that  a  true  population  value  may  be  rejected  by  a  statistical  test.  In 
the  case  of  the  two  surveys,  Battelle  used  a  95-percent  and  EDS  used  a  75-percent  probability  that  the 
differences  in  relocation  satisfaction  levels  were  due  to  something  other  than  random  occurrence. 
Battelle  found  no  significant  difference  in  satisfaction  levels,  with  a  5-percent  chance  of  reaching  an 
incorrect  conclusion.  EDS  did  find  a  significant  difference  but  with  a  25-percent  chance  of  reaching  an 
incorrect  conclusion. 
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Limited  Data  on  Claims 
Settlement  Time 


satisfaction  and  highlight  areas  that  could  be  addressed  in  future 
evaluations.  For  example,  the  Battelle  survey  found  that  61  percent  of  the 
Hunter  respondents  were  more  satisfied  with  their  most  recent  (pilot) 
moving  experience  than  with  their  prior  move.  Eighty-seven  percent  of 
these  respondents  were  satisfied  or  very  satisfied  with  the  carrier’s 
responsiveness.  Both  surveys  also  indicated  that  servicemembers 
relocating  within  the  continental  United  States  were  generally  more 
satisfied  with  the  pilot  process  than  were  personnel  moving  overseas. 

The  surveys  also  showed  that  servicemembers  liked  some  unique  features 
the  pilot  offered  such  as  one-on-one  counseling  and  the  simplified  claims 
process.  However,  few  servicemembers  took  advantage  of  many  of  the 
additional  services  that  Cendant  Mobility  offered,  particularly  assistance  in 
house  hunting.  Cendant  Mobility  reported  to  us  that  its  referrals  to  real 
estate  companies  resulted  in  the  sale  of  one  residence  in  the  Hunter  area 
and  the  purchase  of  five  residences  at  new  duty  stations.  Moreover, 
officials  of  The  Military  Coalition,  an  association  that  represents  5  million 
active  duty,  retired,  reserve,  and  National  Guard  personnel  and  their 
families,  told  us  that  more  could  be  done  to  tailor  relocation  services  to 
meet  unique  militaiy  needs.  Such  services  include  assistance  in  finding 
rental  property,  because  military  personnel  tend  to  rent  rather  than 
purchase,  and  assistance  for  spouses  seeking  employment. 

One  of  the  lessons  learned  is  related  to  access  to  survey  data  by  outside 
reviewers.  AAA  was  unable  to  compare  respondent  results  across  surveys 
because  the  survey  contractors,  citing  privacy  act  restrictions,  did  not  grant 
access  to  identifier  documentation.  The  Army  stated  that  in  future  pilot 
projects  it  would  ensure  that  servicemembers  are  not  subjected  to  multiple 
surveys  and  would  establish  a  better  survey  method  for  determining 
customer  satisfaction. 

Due  to  weaknesses  in  the  Army’s  data  collection  and  analysis,  we  could  not 
validate  that  the  initial  offer  to  settle  claims  took  an  average  of  9  days  after 
a  completed  claim  form  was  received  by  Cendant  Mobility.  Furthermore, 
the  Army’s  methodology  was  limited  to  examining  the  time  between  filing  a 
completed  claim  form  and  making  the  initial  settlement  offer.  This  measure 
does  not  capture  the  entire  claims  process  and  therefore  does  not  measure 
the  extent  to  which  it  represents  an  improvement  over  the  existing  system. 
The  pilot  process,  however,  does  offer  benefits  to  the  servicemember  in 
terms  of  full  replacement  value  protection.  Household  effects  are  insured 
for  their  full  replacement  value  (up  to  $75,000)  rather  than  their 
depreciated  value  (which  is  capped  at  $40,000  under  the  existing  system). 
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The  pilot  process  also  requires  less  work  from  the  servicemember  because 
the  contractor  provides  estimates  on  lost  and  damaged  household  goods 
and  pays  the  servicemember  directly. 

The  claims  settlement  time  measurement  is  based  on  the  contractual 
requirement  that  Cendant  Mobility  offer  settlement  to  a  member  within 
30  days  after  receiving  a  claim  for  damaged  or  lost  property.  Consequently, 
the  Army  developed  a  threshold  for  claims  settlement  of  30  days.  Under  the 
Hunter  pilot,  servicemembers  could  appeal  to  the  contractor  for 
reconsideration  by  providing  additional  information  or  file  a  claim  with  the 
Army  if  resolution  with  the  contractor  could  not  be  achieved — a  process 
that  could  take  more  than  2  years  to  complete.  While  we  found  that  some 
claims  remained  unpaid  and  unsettled  because  the  claims  were  put  “on 
hold"  by  the  servicemember,  the  Army  reported  that  no  one  in  the  Hunter 
pilot  filed  a  claim  with  the  Army  claims  office  as  of  January  1999. 

The  Army  reported  that  Cendant  Mobility  took  an  average  of  9  days  to  offer 
settlement  for  lost  or  damaged  property,  but  neither  AAA  nor  we  could 
confirm  this  because  the  Army  had  not  collected  the  necessary  data.  AAA 
determined,  and  we  concur,  that  Cendant  Mobility  may  have  taken  as  long 
as  an  average  of  28  days  to  offer  settlement.  This  figure  was  based  on  the 
time  between  the  date  the  claim  was  signed  and  the  date  the  member  was 
offered  settlement.  The  difference  between  the  two  estimates  was 
primarily  due  to  a  19-day  period  between  the  date  the  claim  was  signed  and 
the  date  the  contractor  reported  entering  the  claim  into  its  database. 
Although  Cendant  Mobility  officials  stated  that  standard  company  practice 
is  to  enter  a  claim  into  the  database  the  same  day  it  is  received,  they  said 
delays  could  be  caused  by  (1)  the  form  being  mailed  some  time  after  it  is 
dated,  (2)  mail  delays,  and  (3)  contractor  delays  in  entering  the  claim  into 
the  database.  Army  officials  stated  that,  due  to  the  difficulties  in  measuring 
claims  settlement  time,  they  would  change  their  data  collection  and 
analysis  of  this  sub-factor  in  future  pilot  programs. 

We  could  not  confirm  that  all  eligible  direct  deliveries  were  made  without 
in-transit  storage  because  the  necessary  data  was  not  collected.  The  Army’s 
methodology  for  this  sub-factor  also  did  not  allow  an  assessment  of 
whether  the  results  represented  an  improvement  over  the  current  process. 
Direct  delivery  is  defined  by  the  Army  as  delivery  of  a  shipment  to  a 
servicemember’s  residence  without  storage  in  transit.  A  shipment  is  eligible 
for  direct  delivery  only  if  it  has  a  destination  address  before  the  shipment  is 
offered  for  delivery.  Direct  delivery  is  a  contract  performance 
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measurement,  and  the  contractor  is  paid  an  incentive  for  maintaining  a 
direct  delivery  rate  of  over  60  percent.  Consequently,  the  Army  threshold 
for  this  measurement  is  60  percent. 

Neither  AAA  nor  we  could  confirm  that  shipments  delivered  directly  to 
servicemembers  were  the  only  ones  eligible  for  direct  delivery  because  the 
necessary  information  on  all  shipments  was  not  collected.  That  is,  other 
shipments  may  have  been  eligible  under  the  Army  criteria;  however,  AAA 
and  we  could  not  determine  which  shipments  had  addresses  prior  to  being 
offered  for  delivery,  and  were  thus  eligible,  but  ended  up  requiring 
in-transit  storage.  The  Army  acknowledged  the  difficulty  in  validating 
eligible  direct  deliveries  and  stated  that,  in  the  future,  it  would  consider 
using  storage-in-transit  rates  as  a  test  factor.  In  the  February  5, 1999, 
supplemental  information  provided  to  us,  the  Army  stated  that  74  percent 
of  Army-wide  shipments  required  in-transit  storage,  compared  with 
66  percent  requiring  temporary  storage  during  the  Army  pilot. 

Servicemembers  benefit  from  having  household  goods  delivered  directly  to 
their  home  because  the  practice  limits  additional  handling  of  their  property, 
reducing  opportunities  for  loss  and  damage.  Also,  the  government  avoids 
temporary  in-transit  storage,  which  is  costly  and  hard  for  the  Army  to 
control.  Household  goods  shipments  require  temporary  storage  when 
servicemembers  do  not  know  their  new  address  at  the  time  of  shipment, 
and/or  they  move  to  their  new  duty  station  after  completing  vacations  or 
military  requirements  en  route. 

Some  Overhead  Cost  Data  Was  Due  to  weaknesses  in  the  Army’s  methodology  for  collecting  overhead  cost 

Inaccurate  and  Incomplete  data  and  the  questionable  reliability  of  some  of  the  overhead  cost  data  that 

was  collected,  we  could  not  validate  that  the  additional  cost  of  the  pilot 
project  was  18.6  percent  higher  than  the  baseline  cost.  As  we  reported  in 
June  1998,  developing  overhead  costs  has  historically  been  difficult  for  the 
government,  including  DOD,  because  such  data  is  often  unreliable  and 
unavailable.  (See  appendix  III  for  a  detailed  breakdown  of  pilot  and 
baseline  costs  as  reported  by  the  Army  and  validated  by  AAA). 

Neither  AAA  nor  we  could  validate  some  overhead  costs.  Specifically,  we 
could  not  confirm  the  accuracy  of  the  overhead  costs  attributed  to  MTMC 
because  the  Army  used  data  from  fiscal  year  1994.  MTMC  has  changed 
significantly  since  then  due  to  downsizing,  but  AAA  could  not  obtain 
updated  costs  from  MTMC  representing  the  current  organization  to 
determine  the  reasonableness  of  this  overhead  cost.  AAA  and  we  also  agree 


Page  9 


GAO/NSI  AD-99  129  Defense  Transportation 


B-282406 


that  a  drop  in  document  processing  (for  claims,  invoices,  and  inbound 
shipments)  would  not  result  in  cost  savings  to  the  government  unless  the 
activities  that  perform  these  functions,  such  as  the  Defense  Finance  and 
Accounting  Service,  take  corresponding  actions  to  reduce  their  costs 
(for  example,  by  reducing  personnel  to  reflect  the  workload  reduction). 

AAA  and  we  also  identified  other  costs  not  included  in  the  Army’s  analysis 
such  as  those  related  to  awarding  the  Hunter  contract  and  the  contractor’s 
use  of  foreign-flag  vessels.8  The  Army  could  not  provide  an  estimated 
dollar  value  associated  with  the  cost  of  developing  the  contract 
solicitation,  reviewing  offers,  and  resolving  bid  protests.  While  we 
recognize  that  some  of  these  estimated  costs  may  be  infrequent  or  one-time 
only,  they  should  be  considered.  Also,  the  Army  did  not  include  the 
contractor’s  use  of  foreign-flag  vessels  into  the  pilot  project’s 
transportation  costs.  While  individually,  costs  such  as  these  are  probably  of 
low-dollar  value  in  relation  to  other  costs  in  the  analysis,  collectively,  they 
could  materially  affect  the  difference  between  the  baseline  and  pilot  costs. 

According  to  the  Army,  some  of  the  reasons  for  the  18.6  percent  higher  cost 
of  the  pilot  project  can  be  attributed  to  (1)  difficulty  in  calculating  accurate 
baseline  costs,  which  the  Army  believes  were  understated;  (2)  low  volume 
of  moves  (1,349) ,  which  did  not  provide  enough  leverage  to  negotiate  better 
rates  and  discounts;  (3)  higher  than  expected  costs  of  overseas  shipments; 
(4)  relatively  high  unaccompanied  baggage  shipment  rates,  which  could 
have  been  lower  using  negotiated  rates;  (5)  packing  allowance  for  DITY 
moves,  which  was  calculated  using  commercial  rates  during  part  of  the 
12-month  test  period,  resulting  in  a  higher  rate;  and  (6)  quality  moving 
services,  which  cost  more.  In  regard  to  the  high  cost  of  overseas  shipments, 
the  Army  provided  us  with  additional  details  of  pilot  project  costs  that 
showed  a  44-percent  increase  for  overseas  household  goods  shipments  and 
a  2-percent  increase  for  domestic  shipments  over  the  baseline  cost. 

Both  AAA  and  we  validated  that  Cendant  Mobility  awarded  33  percent  of 
the  shipments  to  small  businesses — 10  percent  to  carriers  and  23  percent  to 
agents.  The  Army  measured  the  impact  of  the  pilot  on  these  two  types  of 
providers  and  established  a  threshold  of  23  percent  to  demonstrate 
successful  participation  by  small  businesses.9  The  Army  based  this  factor 
on  section  15(g)  of  the  Small  Business  Act,  which  establishes  a 


Small  Business  Usage  Exceeded 
Army  Goals 


8Foreign-flag  vessels  are  ships  registered  in  foreign  countries. 

9In  the  motor  freight  and  transportation  industry,  firms  with  annual  gross  revenues  of  $18.5  million  or 
less  are  classified  as  small  businesses.  See  Federal  Acquisition  Regulation  19.102. 
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governmentwide  goal  for  participation  by  small  business  concerns  at  not 
less  than  23  percent  of  the  value  of  all  prime  contracts.  The  Army  reported 
that  261  (33  percent)  of  the  790  Hunter  shipments  billed  during  the 
12-month  pilot  were  subcontracted  by  Cendant  Mobility  to  small  carriers 
and  agents. 

As  we  reported  in  June  1998,  Cendant  Mobility  encountered  problems  in 
attracting  small  carriers  and  agents  to  participate  in  the  Hunter  pilot.  When 
the  pilot  was  implemented,  several  local  agents  and  the  companies  they 
represented  elected  not  to  sign  agreements  to  participate.  Cendant  Mobility 
reported  that  historically,  over  40  percent  of  the  contracts  it  awards  go  to 
small  and  small  and  disadvantaged  businesses.  However,  as  of 
December  31, 1997,  only  14  percent  of  the  shipments  booked  by  Cendant 
Mobility  during  the  first  6  months  had  been  awarded  to  small  businesses. 
By  June  30, 1998,  a  number  of  national  van  lines  and  their  local  agents  had 
reversed  their  decision  and  had  signed  contracts  with  Cendant  Mobility. 


No  Comprehensive 
Strategy  for  Evaluating 
Multiple  Pilots 


DOD  will  be  running  multiple  pilots  concurrently,  each  with  different  goals 
and  objectives  and  expected  outcomes.  MTMC  reengineering  efforts  began 
in  1994  followed  by  the  Army’s  in  1996  and  the  Navy’s  in  1997.  DOD  is 
proposing  a  fourth  pilot  with  no  specific  start  date.  While  USTRANSCOM  is 
tasked  with  evaluating  the  results  of  these  pilots  and  using  the  information 
to  recommend  a  redesigned  Department-wide  relocation  program,  it  does 
not  have  a  comprehensive  strategy  for  doing  so.  Specifically,  it  has  not  yet 
determined  how  many  approaches  it  will  ultimately  test,  it  has  not  set  the 
milestones  for  completing  the  pilots’  evaluations  and  implementing  an 
improved  process,  and  it  has  not  assured  itself  that  a  methodologically 
sound  evaluation  process  is  in  place  to  execute  this  process. 


MTMC’s  Reengineering 
Efforts  Began  in  1994 


DOD  first  proposed  reengineering  its  personal  property  program  in  1994. 
Specifically,  on  June  21, 1994,  USTRANSCOM  directed  MTMC  (the  Army 
component  of  USTRANSCOM  and  the  program  manager  for  DOD's 
Personal  Property  Shipment  and  Storage  Program)  to  reengineer  the 
personal  property  program.  On  June  15,  1995,  the  House  Committee  on 
National  Security,10  concurring  that  DOD  must  pursue  a  higher  level  of 
service,  directed  that  DOD  undertake  a  pilot  program  to  implement 
commercial  business  practices  and  standards  of  service.  MTMC  planned  to 


10Now  called  the  House  Committee  on  Armed  Services. 
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award  contracts  for  the  new  program  pursuant  to  the  competitive 
acquisition  system  provisions  (10  U.S.C.  ch.  137  and  its  implementing 
regulation  in  the  Federal  Acquisition  Regulation).  Expressing 
congressional  concerns  about  the  impact  the  competition  system  and  any 
nonstandard  commercial  business  practice  requirements  might  have  on 
small  businesses,  the  statement  of  managers  in  the  conference  report 
accompanying  the  1997  DOD  Authorization  Act  directed  that  DOD  report 
on  the  impact  of  the  pilot  program  on  small  businesses.  After  reviewing  the 
reports,  the  House  Committee  on  National  Security  remained  concerned 
that  MTMC’s  pilot  program  did  not  satisfactorily  address  issues  raised  by 
small  moving  companies  and  directed  that  the  Secretaiy  of  Defense 
establish  a  working  group  of  military  and  industry  representatives  to 
develop  an  alternative  pilot  program. 

Although  the  working  group  reached  consensus  on  many  issues,  including 
a  set  of  program  goals,  it  could  not  reach  agreement  on  the  approach  to 
take  for  the  pilot  test.  Consequently,  the  two  sides  presented  separate 
proposals.  In  November  1996,  we  reported  that  in  our  assessment,  MTMC’s 
proposal  met  the  goals  of  reengineering  the  personal  property  program  to  a 
greater  extent  than  the  industry’s  plan.  1 

The  MTMC  pilot  program,  delayed  by  numerous  bid  protests,  finally  started 
in  January  1999. 12  The  pilot  includes  50  percent  of  the  moves  originating  in 
North  Carolina,  South  Carolina,  and  Florida.  The  pilot  will  run  concurrently 
with  the  existing  MTMC-managed  program  at  installations  representing  all 
military  services  and  the  Coast  Guard  in  the  three  states  and  will  involve 
approximately  18,500  annual  moves.  The  key  features  of  the  pilot  program 
include  selecting  carriers  based  on  servicemember  satisfaction  and  past 
performance  rather  than  simply  price;  achieving  stronger  carrier 
commitment  with  long-term  contracts;  and  offering  full  replacement  value 
protection  and  direct  claims  settlement  to  users.  MTMC’s  pilot  will  run  for 
a  3-year  test  period  (1  year  and  two  1-year  option  periods)  which  will  end  in 
December  2002. 


11Defense  Transportation:  Reengineering  the  DOD  Personal  Property  Program  (GA0/NSIAD-97-49, 
Nov.  27,  1996). 

^Solicitation  disputes  in  the  following  decisions:  Aalco  Forwarding,  Inc.,  et  al.  B-277241.8,  B-277241.9, 
October  21,  1997;  Aalco  Forwarding.  Inc,  et  al..  B-277241.12,  B-277241.13,  December  29,1997; 

Aalco  Forwarding.  Inc,  et  al..  B-277241.15,  March  11, 1998;  Aalco  Forwarding,  Inc,  et  al..  B-277241.16, 
March  11,  1998;  Aalco  Forwarding.  Inc.,  et  al..  B-277241,  277241.14,  June  8,  1998;  and 
Aalco  Forwarding.  Inc.,  et  al..  B-277241,  B-277241.20,  B-277241.21,  July  1, 1998. 
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Hunter  Pilot  Effort  Began  in  Separately  from  the  MTMC  pilot  program,  the  Army  decided  to  determine 
1996  whether  the  commercial  business  practice  of  outsourcing  a  non-core 

function  could  alleviate  known  problems  in  its  personal  property  program 
while  increasing  quality  of  life.  The  pilot  began  in  February  1996  as  a 
quality-of-life  effort  to  improve  the  relocation  process  and  test  commercial 
business  practices  in  a  military  environment  at  Hunter  Army  Airfield, 
Savannah,  Georgia.  On  January  31, 1997,  the  Army  selected  PHH 
Relocation,  a  relocation/move  management  company  (now  Cendant 
Mobility,  Bethesda,  Maryland) ,  as  the  contractor  for  the  pilot.  In  July  1997, 
Cendant  Mobility  began  relocation  operations  at  Hunter,  offering  to 
distribute  and  manage  shipments  to  transportation  providers.  Additionally, 
the  contractor  provided  point-to-point  move  management,  personal  move 
counseling  and  coordination,  assistance  in  buying/selling  a  residence, 
24-hour  in-transit  visibility  of  shipments,  direct  claims  settlement  with 
servicemembers,  full  replacement  value,  and  a  single  point  of  contact  for 
servicemembers.  These  services  and  benefits  are  not  available  under  the 
existing  DOD  system.  The  current  contract  with  Cendant  Mobility  will  end 
on  September  30, 1999. 

A  Cendant  Mobility  official  stated  that  the  management  fees  paid  by  the 
Army  to  the  contractor  pay  for  these  services  and  entitlements.  Cendant 
Mobility  also  obtained  revenues  from  two  other  sources,  including 

(1)  commissions  from  carriers,  agents,  and  freight  forwarders  and 

(2)  referral  fees  from  real  estate  companies  on  the  purchase  and  sale  of  six 
residences.  The  commissions  paid  for  services  provided  to  carriers,  agents, 
and  freight  forwarders,  including  contract  training,  contract 
administration,  invoice  auditing,  and  quality  control  services. 


The  Navy  initiated  its  plans  for  a  separate  pilot  program  in  1997  to  test  the 
option  of  allowing  servicemembers  to  select  their  carrier,  giving  them  more 
control  over  the  relocation  process  to  meet  their  specific  needs.  The  pilot, 
which  began  in  January  1998,  is  intended  to  provide  an  alternative  to, 
rather  than  a  replacement  of,  the  current  program.  The  pilot  affects  a  very 
small  number  of  military  personnel.  Presently,  the  option  is  being  offered 
only  for  shipments  originating  in  the  areas  of  Puget  Sound,  Washington; 
San  Diego,  California;  Norfolk,  Virginia;  and  Groton,  Connecticut. 
Participation  is  limited  to  Navy  military  personnel  with  permanent  change 
of  station  orders  from  the  participating  sites.  Shipments  must  exceed 
3,000  pounds  and  cost  between  $2,500  and  $25,000  (the  pilot  excludes 
shipments  from  non-temporary  storage  or  warehouses).  Servicemembers 


Navy  Pilot  Effort  Began  in 
1997 


Page  13 


GAO/NSIAD-99-129  Defense  Transportation 


B-282406 


can  select  a  mover  from  a  list  of  carriers  that  are  self-certified  as  small 
businesses  and  approved  by  MTMC.  As  of  March  26, 1999,  223  Navy 
servicemembers  had  selected  this  option.  This  pilot  does  not  have  a 
specific  end  date. 


DOD  Pilot  Effort  Proposed  On  February  12, 1999,  DOD  stated  that  it  intended  to  begin  a  fourth  pilot 
in  1999  program,  called  the  Full  Service  Moving  Project,  with  certain  modifications 

based  on  lessons  learned  from  the  Army  Hunter  pilot.  The  new  pilot  will 
include  the  National  Capital  Region  (17,000  annual  moves),  Georgia 
(26,000  annual  moves) ,  and  North  Dakota  (2,000  annual  moves),  and  will 
involve  all  military  services  and  the  Coast  Guard.  Like  the  Army  pilot,  it  is 
intended  to  outsource  a  non-core  function  and  alleviate  known  problems  in 
the  DOD  personal  property  program  while  improving  the  quality  of  life 
associated  with  the  relocation  of  servicemembers  and  their  families.  While 
it  is  projected  to  begin  in  calendar  year  1999,  presently,  this  pilot  does  not 
have  an  official  start  date. 


Unresolved  Issues  in  USTRANSCOM  is  in  the  process  of  finalizing  its  evaluation  plan  for  the 

Evaluating  Pilot  Programs  ongoing  and  planned  pilots.  It  is  proposing  to  evaluate  the  pilots  on  the 

&  °  same  three  factors  that  the  Army  used  in  its  evaluation  of  the  Hunter  pilot, 

except  that  the  factors  would  be  defined  differently  and  would  include  a 
more  expansive  point  scale.  Unresolved  issues,  as  of  February  1999, 
included  the  development  of  a  method  that  would  recognize  the  unique 
characteristics  and/or  process  improvements  of  each  pilot  program  and  the 
validation  of  the  baseline  indirect  costs  that  would  be  used  for  each  pilot 
program.  The  evaluation  plan  does  not  currently  include  an  evaluation  of 
the  Hunter  pilot,  but  USTRANSCOM  used  lessons  learned  in  the  Hunter 
pilot  to  develop  the  plan.  We  provided  comments  to  USTRANSCOM  in  this 
process  and,  among  other  things,  encouraged  the  command  to  seek  expert 
methodological  advice  before  finalizing  the  evaluation  plan  to  enhance  the 
quality  of  its  assessment. 


Conclusions  We  support  the  use  of  pilots  as  a  means  to  test  new  concepts.  Although  this 

process  has  been  ongoing  since  the  early  1990s,  DOD  is  not  yet  in  a  position 
to  determine  what  changes  it  needs  to  make  to  the  process.  Also,  thus  far, 
DOD  has  not  decided  how  many  approaches  it  will  ultimately  test  and  has 
not  developed  evaluation  plans  that  will  yield  accurate  and  credible  data. 
Lessons  learned  from  the  Hunter  pilot  do  provide  useful  information  for 
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evaluating  the  three  other  pilots  ongoing  and  planned.  While 
USTRANSCOM  is  in  the  process  of  developing  a  strategy  to  conduct  this 
evaluation,  it  has  not  yet  resolved  a  number  of  issues,  including  timelines 
for  completing  the  analysis  and  implementing  an  improved  process  and  a 
methodology  for  the  evaluation. 

Recommendations 

We  recommend  that  the  Secretary  of  Defense  direct  the  Commander  in 
Chief,  USTRANSCOM,  to  develop  a  comprehensive  strategy  for  evaluating 
each  of  the  pilot  approaches  that  includes  (1)  tests  of  a  finite  number  of 
approaches;  (2)  timelines  for  completing  the  analysis  and  for  implementing 
a  new  personal  property  process;  and  (3)  milestones  for  finalizing  a 
methodologically  sound  evaluation  plan,  including  specific  criteria  for 
assessing  each  pilot’s  attributes  and  validating  the  baseline  indirect  costs  to 
be  used  for  each  pilot  program.  To  facilitate  this  process  and  to  enhance 
the  quality  of  the  assessment,  we  also  recommend  that  DOD  seek  expert 
methodological  advice  before  finalizing  the  evaluation  plan. 

Agency  Comments  and 
Our  Evaluation 

In  written  comments  on  a  draft  of  this  report,  DOD  stated  that  it  generally 
concurred  with  the  report  and  its  recommendations.  DOD  tasked 
USTRANSCOM  to  evaluate  the  results  of  all  the  pilots  and  provide  a 
recommendation  for  a  DOD-wide  program  to  implement  for  the  long  term. 
DOD  stated  that  USTRANSCOM  is  in  the  process  of  retaining  a  contractor 
to  independently  validate  success  criteria  for  a  reengineered  DOD  personal 
property  program,  develop  a  comprehensive  strategy  for  evaluating  each  of 
the  pilot  approaches,  and  support  the  development  of  recommendations 
and  alternative  courses  of  action.  USTRANSCOM  anticipates  having  the 
contractor  commence  its  efforts  by  June  1999. 

DOD’s  comments  are  presented  in  appendix  IV.  DOD  also  provided 
technical  comments  and  we  revised  our  report  to  reflect  them  where 
appropriate. 

Scope  and 

Methodology 

To  validate  the  results  and  savings  of  the  Hunter  pilot,  we  reviewed  the 
Army’s  plans  to  determine  how  it  would  evaluate  the  pilot’s  results.  We  also 
reviewed  the  Army’s  October  1998  report  and  the  February  and  March  1999 
supplements  detailing  the  12-month  test  results.  We  visited  both  Hunter 
Army  Airfield  and  the  transportation  division  at  Fort  Stewart,  Georgia.  We 
contacted  Hunter  contractor  officials  and  representatives  from  the  two 
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survey  firms  hired  by  the  Army  and  USTRANSCOM  to  measure  customer 
satisfaction.  We  also  examined  the  documents  and  data  compiled  by  the 
AAA  to  validate  the  accuracy  of  the  Hunter  results. 

We  met  with  Army,  Navy,  MTMC,  and  USTRANSCOM  officials  and 
reviewed  documents  to  determine  the  status  and  goals  of  other  DOD 
personal  property  pilot  programs.  We  also  obtained  statistics  on  actual  use 
by  servicemembers  of  the  Navy’s  pilot  as  of  March  1999.  We  did  not  request 
statistics  on  other  pilots  because  the  MTMC  pilots  began  in  January  1999 
and  the  DOD-proposed  Full  Service  Moving  Project  is  still  in  the  planning 
stage.  We  also  met  with  USTRANSCOM  officials  and  reviewed  evaluation 
plans  to  determine  how  DOD  proposes  to  make  changes  to  its  current 
personal  property  program  using  lessons  learned  from  the  pilots. 

We  met  and  discussed  matters  related  to  DOD  pilots  and  efforts  to  improve 
the  existing  personal  property  program  with  officials  from  the  Office  of  the 
Deputy  Under  Secretary  of  Defense  (Logistics-Transportation  Policy) , 
Washington,  D.C.;  USTRANSCOM,  Scott  Air  Force  Base,  Illinois;  the  Office 
of  the  Army  Deputy  Chief  of  Staff  for  Logistics,  Washington,  D.C.;  MTMC, 
Falls  Church,  Virginia;  the  Naval  Supply  Systems  Command, 
Mechanicsburg,  Pennsylvania;  Hunter  Army  Airfield,  Savannah,  Georgia; 
and  Fort  Stewart,  Georgia. 

In  addition,  we  discussed  the  survey  results  with  officials  from  Battelle’s 
Pacific  Northwest  National  Laboratory  and  from  Electronic  Data  Systems. 

We  conducted  our  review  between  August  1998  and  April  1999  in 
accordance  with  generally  accepted  government  auditing  standards. 


We  are  sending  copies  of  this  report  to  the  Honorable  William  S.  Cohen, 
Secretary  of  Defense;  the  Honorable  Louis  Caldera,  Secretary  of  the  Army; 
General  Charles  T.  Robertson,  Jr.,  Commander  in  Chief,  USTRANSCOM; 
Lieutenant  General  John  G.  Cobum,  Deputy  Chief  of  Staff,  Logistics, 
Department  of  the  Army;  Major  General  Mario  F.  Montero,  Jr.,  Commander, 
MTMC;  and  the  Honorable  Jacob  J.  Lew,  Director,  Office  of  Management 
and  Budget.  We  are  also  sending  copies  to  other  congressional  members 
who  expressed  interest  to  us  in  DOD’s  efforts  to  improve  its  personal 
property  program,  including  the  Honorable  Fred  Thompson,  United  States 
Senate,  and  the  Honorable  Neil  Abercrombie,  Duncan  Hunter,  Martin  T. 
Meehan,  and  James  M.  Talent,  House  of  Representatives.  We  will  also  make 
copies  available  to  others  upon  request. 
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Please  contact  me  on  (202)  512-8412  if  you  or  your  staff  have  questions 
concerning  this  report.  Major  contributors  to  this  report  are  listed  in 
appendix  V. 


David  R.  Warren,  Director 
Defense  Management  Issues 
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List  of  Congressional  Committees 

The  Honorable  John  W.  Warner 
Chairman 

The  Honorable  Carl  Levin 
Ranking  Minority  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Ted  Stevens 
Chairman 

The  Honorable  Daniel  K.  Inouye 
Ranking  Minority  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Floyd  D.  Spence 
Chairman 

The  Honorable  Ike  Skelton 
Ranking  Minority  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Herbert  H.  Bateman 
Chairman 

The  Honorable  Solomon  P.  Ortiz 
Ranking  Minority  Member 
Subcommittee  on  Military  Readiness 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Jerry  Lewis 
Chairman 

The  Honorable  John  P.  Murtha 
Ranking  Minority  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Thresholds  and  Scores  Used  by  the  Army  for 
Hunter  Pilot  Evaluation  Factors 


Factors/definitions 

Thresholds 

Scores 

Quality  of  life/customer  satisfaction  rating:  percent  of  Hunter  respondents  who  were 
more  satisfied  than  Fort  Stewart  respondents  (control  group). 

25%  and  above 

1.00  point 

20-24% 

0.75  points 

15-19% 

0.50  points 

10-14% 

0.25  points 

Below  10% 

Unsuccessful 

Quality  of  life/claims  settlement  time:  time  period  between  when  a  claim  was  received 
and  when  the  contractor  offered  settlement. 

0-10  days 

1 .00  point 

11-15  days 

0.75  points 

16-20  days 

0.50  points 

20-30  days 

0.25  points 

Above  30  days 

Unsuccessful 

Quality  of  life/percentage  of  eligible  deliveries:  percent  of  shipments  delivered  to 
residence  without  storage  in-transit. 

91-100% 

1 .00  point 

81-90% 

0.75  points 

71-80% 

0.50  points 

61-70% 

0.25  points 

60%  and  below 

Unsuccessful 

Total  cost:  extent  total  contract  cost  exceeded  total  baseline  cost. 

0% 

1 .00  point 

1-10% 

0.75  points 

11-20% 

0.50  points 

21-30% 

0.25  points 

More  than  30  % 

Unsuccessful 

Impact  on  small  business:  percent  of  small  agents  and  carriers  used. 

20%  or  more 

1.00  point 

27-28% 

0.75  points 

25-26% 

0.50  points 

23-24% 

0.25  points 

Less  than  23% 

Unsuccessful 
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Types  and  Number  of  Shipments  Made  During 
the  Hunter  Pilot  Test 


Shipment  type 

Shipments  initiated  and  recorded  as  completed 

Household  goods  (intrastate  -  Georgia 

51 

Household  goods  (interstate -continental  U.S.,  including  Alaska) 

418 

Household  goods  (overseas  -  Hawaii) 

29 

Household  goods  (overseas  -  Germany) 

79 

Household  goods  (overseas  -  Korea) 

50 

Household  goods  (overseas -other) 

47 

Unaccompanied  baggage  (overseas  -  all  locations) 

378 

Do-it-yourself  (DITY)  moves 

292 

Mobile  homes 

5 

Total 

1,349 
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Hunter  Pilot  and  Baseline  Costs  Calculated  by 
the  Department  of  the  Army  (Office  of  the 
Deputy  Chief  of  Staff  for  Logistics) 


Cost  elements3 

Estimated  pilot  costs 

Estimated  baseline  costs 

Transportation  costs 

Transportation 

$3,005,229 

$2,102,384 

Accessorials 

25,767 

30,655 

Storage 

210,706 

211,655 

Total  transportation  costs 

$3,241,702 

$2,344,694 

Claims  paid  by  Army 

Claims  paid  by  Army _ $0 _ $66,268 


Overhead  costs 

Personnel 

$79,802 

$221,721 

Management  price 

132,700 

0 

MTMC  headquartersb 

0 

63,510 

Automation 

0 

52,612 

Voucher  processing 

4,614 

90,658 

Inbound  processing 

0 

46,834 

Claims  processing 

0 

11,510 

Pay  for  performance 

14,756 

0 

Building  overhead 

28,321 

43,286 

Telephone  and  copier 

1,281 

10,106 

Consumables 

3,760 

5,189 

Total  overhead  costs 

$265,234 

$545,426 

Total  costs 

$3,506,936 

$2,956,388 

Pilot 

costs 

Baseline 

costs 

Difference 

Percent 

difference 

Percent  pilot  costs 
exceeded  baseline  costs 

$3,506,936 

$2,956,388 

$550,548 

18.6 

aArmy  Audit  Agency  validated  all  costs  except  those  under  $5,000,  concluding  that  such  amounts  were 
immaterial. 

b Military  Traffic  Management  Command  (MTMC). 

Source:  Office  of  the  Deputy  Chief  of  Staff  for  Logistics,  Department  of  Army. 
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Comments  From  the  Department  of  Defense 


Note;  GAO  comment 
supplementing  those  in  the 
report  text  appear  at  the  end 
of  this  appendix. 


See  comment  1. 


ACQUISITION  AND 
TECHNOLOGY 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3000 

MAY  I  I  1999 


Mr.  David  R.  Warren 

Director,  Defense  Management  Issues 

U.S.  General  Accounting  Office 

National  Security  and  International  affairs  Division 

Washington,  DC  20548 


Dear  Mr.  Warren: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  report  DEFENSE  TRANSPORTATION:  Hunter  Pilot  Is  Inconclusive  But  Provides 
Lessons  for  Evaluating  Other  Pilots,'1  dated  April  26,  1999  (GAO  Code  709365),  OSD  Case 
1800. 


The  DoD  concurs,  with  comment,  with  the  report  and  recommendations.  Specific  Army 
comments  as  well  as  technical  comments  on  the  report  are  enclosed. 

The  Department  concurs  with  the  Army  comments  on  the  Hunter  household  goods  test 
results.  GAO  states  they  were  unable  to  validate  all  reported  results  due  to  weaknesses  in  the 
Army's  evaluation  methodology.  Specifically,  they  could  not  confirm  that  customer  satisfaction 
improved  due  to  multiple  surveying.  Additionally,  they  could  not  validate  the  extent  to  which 
pilot  costs  exceeded  baseline  costs.  The  Department  believes  that  additional  conclusions  can  be 
drawn  from  the  Hunter  Pilot  which  provide  the  Department  with  positive  results  which  warrant 
expanded  testing  and  careful  evaluation: 

1)  Although  multiple  surveys  were  administered,  all  survey  results  indicated  high 
levels  of  customer  satisfaction. 

2)  GAO  focused  on  survey  results  of  one  question  and  did  not  adequately  consider 
survey  results  of  other  important  questions  that  yielded  positive  test  results. 

3)  The  bulk  of  the  cost  elements  were  validated,  yet  GAO  indicated  costs  were 
inaccurate  and  incomplete. 

4)  GAO  did  not  acknowledge  a  significant  improvement  to  the  claims  process. 

DoD  concurs  with  GAO's  recommendations  to  develop  a  comprehensive  strategy  for 
evaluating  each  of  the  pilots  and  to  seek  expert  methodological  advice  before  finalizing  the 
evaluation  plan  to  enhance  the  quality  of  its  assessment.  U.S.  Transportation  Command 
(USTRANSCOM)  has  been  tasked  by  DoD  to  evaluate  the  results  of  all  the  pilots  and  provide  a 
recommendation  for  a  DoD- wide  program  to  implement  for  the  long  term.  USTRANSCOM  is 
currently  in  the  process  of  obtaining  the  services  of  an  independent  contractor.  That  contractor 
will  support  USTRANSCOM's  requirement  to  independently  validate  success  criteria  for  a 
reengineered  DoD  personal  property  program,  develop  a  comprehensive  strategy  for  evaluating 
each  of  the  pilot  approaches  using  commonly  accepted  accounting  and  statistical  methodologies. 
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The  following  are  GAO’s  comments  on  the  Department  of  Defense’s  letter 
dated  May  11,  1999. 


GAO  Comment  l-  As  stated  in  the  report,  we  could  not  confirm  the  customer  satisfaction 

survey  results.  Specifically,  we  could  not  confirm  that  all  survey  results 
indicated  higher  levels  of  customer  satisfaction  because  of  weaknesses  in 
the  survey  methodology.  Methodological  weaknesses  that  compromised 
survey  results  included  using  different  populations  and  respondent  rates, 
using  various  survey  instruments  with  different  questions,  and  surveying 
servicemembers  multiple  times.  In  addition,  one  of  the  two  survey 
contractors  found  no  statistically  significant  difference  between  responses 
from  Hunter  and  Fort  Stewart. 

Our  assessment  of  customer  satisfaction  results,  as  indicated  in  the  report, 
was  based  on  the  Army’s  criteria  set  forth  in  its  evaluation  plan.  Because 
the  Army  used  the  results  of  one  question  to  score  the  pilot  a  success,  we 
also  used  it  as  the  basis  for  our  assessment.  We  agree,  however,  that  other 
data  obtained  by  the  surveys  provided  indicators  of  customer  satisfaction 
and  highlighted  areas  that  could  be  addressed  in  future  evaluations. 

We  agree  that  the  bulk  of  cost  elements  could  be  validated  and  revised  our 
report  to  show  that  some  overhead  cost  data  was  inaccurate  and 
incomplete. 

Finally,  regarding  our  assessment  of  the  claims  process,  we  did  not 
acknowledge  a  significant  improvement  in  the  process  because  the  Army 
measured  only  one  aspect  of  it,  as  indicated  in  the  report.  In  addition,  on 
the  basis  of  the  Army’s  criteria  set  forth  in  its  evaluation  plan  for  assessing 
this  factor,  we  could  not  confirm  the  time  it  took  to  offer  claims  settlement 
to  servicemembers.  We  also  agree  that  the  claims  process  offered  benefits 
not  previously  available  to  the  servicemembers. 
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Major  Contributors  to  This  Report 

National  Security  and 
International  Affairs 
Division,  Washington, 
D.C. 

t - — - 

Charles  I.  Patton,  Jr.,  Associate  Director 

Nomi  R.  Taslitt,  Assistant  Director 

Robert  L.  Self,  Evaluator-in-Charge 

Marc  J.  Schwartz,  Senior  Evaluator 

Jacqueline  E.  Snead,  Senior  Evaluator 

Richard  R.  Irving,  Evaluator 

Arthur  L.  James,  Jr.,  Mathematical  Statistician 

Office  of  the  General 
Counsel,  Washington, 
D.C. 

l 

John  G.  Brosnan,  Assistant  General  Counsel 

Norfolk  Field  Office 

r 

Daniel  A.  Omahen,  Senior  Evaluator 

John  R.  Beauchamp,  Senior  Evaluator 
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The  first  copy  of  each  GAO  report  and  testimony  is  free. 

Additional  copies  are  $2  each.  Orders  should  be  sent  to  the 
following  address,  accompanied  by  a  check  or  money  order  made 
out  to  the  Superintendent  of  Documents,  when  necessary,  VISA  and 
MasterCard  credit  cards  are  accepted,  also. 

Orders  for  100  or  more  copies  to  be  mailed  to  a  single  address  are 
discounted  25  percent. 

Orders  by  mail: 

U.S.  General  Accounting  Office 
P.O.  Box  37050 
Washington,  DC  20013 

or  visit: 

Room  1100 

700  4th  St.  NW  (corner  of  4th  and  G  Sts.  NW) 

U.S.  General  Accounting  Office 
Washington,  DC 

Orders  may  also  be  placed  by  calling  (202)  512-6000 

or  by  using  fax  number  (202)  512-6061,  or  TDD  (202)  512-2537. 

Each  day,  GAO  issues  a  list  of  newly  available  reports  and 
testimony.  To  receive  facsimile  copies  of  the  daily  list  or  any  list 
from  the  past  30  days,  please  call  (202)  512-6000  using  a  touchtone 
phone.  A  recorded  menu  will  provide  information  on  how  to  obtain 
these  lists. 

For  information  on  how  to  access  GAO  reports  on  the  INTERNET, 
send  an  e-mail  message  with  “info”  in  the  body  to: 

info@www.gao.gov 

or  visit  GAO’s  World  Wide  Web  Home  Page  at: 
http  ://www.gao  .gov 
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